Load testing methods and systems with transaction variability and consistency

ABSTRACT

Disclosed are load testing methods and systems with transaction variability and consistency.

CROSS REFERENCE TO RELATED APPLICATIONS

This nonprovisional patent application is based upon and claims priorityfrom U.S. provisional patent application Ser. No. 60/418,824, filed Oct.16, 2002, entitled “Load Testing Methods And Systems With TransactionVariability And Consistency.”

This nonprovisional patent application is related to commonly-owned,co-pending U.S. nonprovisional patent application Ser. No. 10/210,798filed Aug. 1, 2002, entitled “Protocol Sleuthing System And Method ForLoad-Testing A Network Server,” set forth in pertinent part below.

FIELD OF THE INVENTION

The present invention is directed generally to network load testingmethods and systems. More particularly, the present invention relates tomethods and systems capable of generating uniquely variable transactioninstances within a set of defined testing populations, thereby providingtransaction variability and consistency, and thus, realistic test loads.

BACKGROUND OF THE INVENTION

The rapidly expanding use of computer networks and application servicesprovided by network servers in today's society has led to an increasedneed to test, monitor, and evaluate the load capacity of such networkservers, particularly in the context of providing application servicesrequested by users.

Client/Server Transactions: In the past, effective load testing ofnetworks and associated application services required a multiplicity ofactual users simultaneously generating continuous transactions with thevarious applications available through a particular network server. Theload represented by these transactions enables network designers,administrators, and operators to analyze the network and applicationservices under stress.

It is impractical, however, to conduct realistic network load testingusing conventional methods. Consider, for example, the hardware andsoftware resources required by a single, actual user in conductingconventional client/server transactions with a network server, asillustrated in FIGS. 1 and 2.

As shown in FIG. 1, the user employs a web browser or similar clientapplication CA running on personal computer PC (or other processor), andan associated graphical user interface GUI, to conduct the client/servertransactions with server application SA on the network server computerNSC. The user also employs the PC's associated hardware, including akeyboard, monitor, and modem or other communications devices, to enterURL addresses for network resources and content.

Similarly, FIG. 2 illustrates a typical client-server transaction(surfing for information on the Internet), using the resourcesillustrated in FIG. 1. The user employs the browser or similar clientapplication CA to request HTML (Hypertext Markup Language) pages fromthe server application SA on the network server computer NSC. Theclient/server transactions consist of a series of defined protocolinteractions between the browser CA and the server application SA. Thesecan include client requests and server responses in accordance with astandard protocol such as FTP (File Transfer Protocol). For example, asshown in FIG. 2, the user initiates the protocol interactions definedfor this particular client-server transaction by request CR1 to connectto the network server computer NSC. The user may do this in theconventional manner by typing a URL address “xyz” of the network servercomputer NSC or clicking on a link. The network server computer NSC thenissues a response SR1, granting or denying the client request. Onceconnected to the network server computer NSC, the client requests thatthe server application SA be opened on a specific port number “abc”(request CR2), and the server NSC responds (SR2) by granting therequest. Next, the client requests a file, www.mysite.com/myfile.html(request CR3), and the server NSC responds by transmitting the requestedfile to the client computer PC. Following client requests CR4 and CR5,and the server's responses SR4, SR5, the client-server transaction isvoluntarily terminated by the user.

It will be readily understood that such client/server transactions areresource intensive. A browser's GUI and display engines requireextensive memory and CPU resources to support the processing andtransmission of application requests and the display of content providedby the server. A single web browser, for example, can utilize as much as10 Megabytes of memory during a single client/server transaction.

Conventional Load Testing: In some systems, network load testing isaccomplished using the client resources and transaction format describedabove, in combination with utility macros that record the user'sinteractions with the client and server applications. These macros canrecord all user input (keystrokes, mouse movements and clicks) andrelevant responses from the server application during the transaction,and the recorded information can be replayed for review and evaluation.However, because the user's interactions with the client and serverapplications are predominately graphical, detecting and isolating errorsis time consuming and labor intensive. Since GUIs do not typicallyinclude elements for automatically detecting error events, it is left tothe user to detect errors visually.

In addition, scaling a conventional load testing method to providestatistically useful load testing data requires generating a largenumber of simulated users. In turn, each simulated user requires theentire client application, including the GUI. Thus, simulating onehundred web users would require the client computer to create onehundred instances of the client application and the network serverapplication. The associated memory and processing requirements couldeasily overwhelm a typical client computer.

Accordingly, there is a significant need for methods and systems thatcan generate a vast number of transactions, to simulate real user loads,without a concomitantly large resource requirement.

It would also be desirable to provide load testing methods and systemscapable of creating traffic that accurately simulates a real user load.In conventional load testing systems, designers attempt to mimic realtraffic by creating multiple populations of (synthetic) users withdefined characteristics. For example, a single user's web browsingbehavior might be defined by the following:

Listing 1:

1. Access web server at IP address xxx. xxx. xxx. xxx.

2. Login as user XYZ with password JKL.

3. Retrieve web page mywebpage.html

4. Wait 15 seconds.

5. Repeat this transaction.

Given this description, a test administrator could create a test bygenerating a population of thousands of instances of this specific userbehavior, thereby creating the user load. However, each instance wouldhave exactly the same attributes. The resulting population of identicalinstances would not create a realistic user load, since real users donot all log into the same server, access the same web page, wait exactly15 seconds and then repeat.

In addition, typical infrastructure, including network devices, HTTPservers, and storage systems, would cache the data, resulting in minimalnetwork traffic, minimal data storage access, and minimal HTTP serveractivity, thus yielding invalid results. A test of this nature wouldfalsely indicate that the network could service thousands more usersthan it actually could under a real load.

The test administrator could create multiple populations, each havingdifferent characteristics. However, this would only be trulyrepresentative of a unique user load if a population were created forevery instance generated. Under such circumstances, it could take thetest administrator weeks to construct a simple test.

Accordingly, there is a significant need for the ability to easilygenerate unique instances of multiple user population classes thatrepresent real loads.

SUMMARY OF THE INVENTION

In one aspect, the invention provides a network load testing systemcomprising: an addressable named list means to enable the generation ofsubstantially random and unique network transaction instances simulativeof real network traffic patterns; addressing means operable to addressthe named list means; and generating means, operable to communicate withthe addressing means, for generating the substantially random and uniquenetwork transaction instances simulative of real network trafficpatterns.

BRIEF DESCRIPTION OF DRAWING FIGURES

FIG. 1 is a block diagram showing a conventional client/servertransaction.

FIG. 2 is a diagram outlining protocol aspects of the transaction ofFIG. 1.

FIG. 3 is a schematic diagram illustrating one practice of the presentinvention.

FIG. 4 is a schematic diagram showing the use of Named List structures.

FIG. 5 is a schematic diagram showing a test in accordance with theinvention.

FIG. 6 is a schematic diagram showing a test plan.

FIG. 7 is a schematic diagram showing an exemplary “screen shot” ofruntime attributes.

FIG. 8 illustrates an error event occurring during the client-servertransaction illustrated in FIG. 2.

FIG. 9 illustrates a protocol sleuthing system.

FIG. 10 is an illustrative example of the implementation of a‘background client’ consisting of four ‘synthetic users’ interactingwith an application available through a network server.

FIG. 11 illustrates the differences between a client-server transactionconducted by a human and a client-server transaction implemented by abackground client via a protocol sleuthing system and method.

DETAILED DESCRIPTION OF THE INVENTION

I. Synthetic Transaction Variability: The present invention generatesuniquely variable transaction instances within a set of defined testingpopulations, thereby providing transaction variability and consistency,and thus, realistic test loads. In one aspect of the invention,synthetic transaction instances, simulative of the network loadpresented by real users, are generated in accordance with a test plancontaining multiple population classes or “groups.” Each group containsattributes that describe the behavior of each instance generated withinthe group. Based on the test plan and the attributes of groups therein,the system generates a number of instances and an appropriate networkprotocol (collectively, the load) for the test. These latter functionsare implemented by a Network Testing Resource (NTR) application.

FIG. 3 illustrates one practice of the present invention, including atest plan with three groups, referred to therein as Population ClassesA, B and C.

Referring to FIG. 3, it will be seen that the system generates, forPopulation Class A, web browser traffic having the attributes shown onthe right-hand side of FIG. 3. In turn, Population Class B could be anFTP (File Transfer Protocol) session with similar attributes, and ClassC could represent a streaming video population.

In another aspect of the invention, described in greater detail below,this information is created by a test editor application, and stored ina configuration file that can be “pushed” to the NTRs, which thengenerate the instances and protocols that constitute the load.

In accordance with FIG. 3 (and using the parameters noted above in theprior art example), the attributes may have the following staticsettings:

Listing 2:

1. IP address=192.168.3.23

2. URL=www.mywebsite.html

3. Userid=John

4. Password=que123

5. SSL Cipher Suite=EXPORT40

6. HTTP Header=German

7. Think time=15

The present invention enables a characteristic referred to herein asSynthetic Transaction Variability (STV), in which each instanceassociated with a user class population has unique attributes. Thisuniqueness, in turn, enables the realistic simulation of actual userloads. One way in which STV is accomplished is through the use of NamedList structures, as shown schematically in FIG. 4.

Referring now to FIG. 4, there is shown a test plan containing a namedlist structure. In one practice of the invention, each test plan has theability to create and store multiple named lists. The named lists arecreated as part of the test plan, and each named list may consist ofURLs, language headers, IP addresses, cipher suites, or any groupattribute that, in the prior art, previously required a static value. Inthe illustrated embodiment, a named list is common to all populationclasses and may be reusable among classes. Each list has a name and anunlimited number of members.

Assume, for example, that a test plan has two lists, one that containsHTTP language headers and another that contains URLs. The first list(Listing 3) is named $listofHTTPHeaders and contains (by way of example)a list of language headers. The second list (Listing 4) is named$listofURLs, and lists candidate URLs, as follows:

Listing 3:

1. German

2. English

3. French

4. Italian

5. Ukrainian

6. etc.

Listing 4:

1. /mywebsite/page1.html

2. /mywebsite/page2.html

3. /mywebsite/page3.html

4. /mywebsite/page4.html

5. /mywebsite/page5.html

6. etc.

Given these list structures, instead of being limited (as was the priorart) to having static attributes that define a population class, theload testing system of the present invention can utilize the lists toprovide Synthetic Transaction Variability. By way of example, the systemcan employ functions that provide random (@lrand) and sequential(@lnext) access to the lists, defined as follows:

@Irand($list)

This function returns a random member of ($list) between 1 andlengthoflist.

@lnext($list)

This function returns the next member of the list ($list). Once the endof the list is reached, the list pointer rewinds to the beginning.

To see how this affects the resulting instances, consider the followingListing 5, showing Listing 2 with STV introduced:

Listing 5:

1. IP address=192.168.3.23

2. URL=@lrand($listofURLs)

3. Userid=John

4. Password=que123

5. SSL Cipher Suite=EXPORT40

6. HTTP Header=@lnext($listofHTTPHeaders)

7. Think time=15

8. Repeat.

Assume that 1000 instances are created for the population class A, inaccordance with Listing 5. Given lines 2 and 6 of Listing 5, eachinstance would get a unique value for URL and HTTP Header. Theseattributes would also vary for each repeat of the loop, thus creatingrandom and realistic user loads.

In the above example (Listing 5), any of the static attributes couldpoint to a list. In this example, a random member from $listofURLs willbe retrieved each time the instance is repeated; and for each iteration,the next member of the list $listofHTTPHeaders will used to generaterequests from different languages. The list pointer will rewind when itreaches the end of the list.

Synthetic Transaction Consistency: The present invention also enablesanother useful property: Synthetic Transaction Consistency or STC. Inaccordance with the invention, STC enables each instance of a user classpopulation to have uniquely predictable runtime attributes. This isadvantageous for designing tests that require authentication or knownparameters within a defined range, with repeatable results.

Consider, for example, a website that requires users to authenticateprior to downloading data. Generating thousands of instances of a userclass would result in the same authentication for each instance, therebyresulting in an improbable scenario. Another example would be a userclass that sends email to another account. Thousands of instances thatgenerate mail sent to one user would not represent real traffic.

Accordingly, the present invention enables synthetic transactionconsistency by using runtime variables to generate unique and consistentattributes for instances of a group. These runtime attributes can bebased upon the following objects used within a test plan:

Test Plan

Resources

Interfaces

Groups

Group Instances

In particular, as schematically shown in FIG. 5, a test consists of atest plan that contains multiple resources, each with one or moreinterfaces that contain one or more groups.

The test plan shown in FIG. 5 contains two resources, each with one(Ethernet) interface that contains two user groups (userload:0 anduserload:1). Thousands of instances can be generated for each group. Inturn, each object has an internal name and numerical value. For example,each resource can be assigned a unique integer 1 through n, where n isthe last resource. This is true for each object in the plan. The testplan can then be represented as shown in FIG. 6.

Runtime Attributes: In one practice of the invention, the systemprovides runtime access to the names and integer values of each testobject. FIG. 7 is an exemplary “screen shot” of the runtime attributes.

The screen shot of FIG. 7 contains a “System Value” on the left, and“Type” on the right. System values in the illustrated example includeTestPlanName, ResourceName, UserName, TransactionName, InterfaceName,ResourceID, InterfaceID, UserID, ID, ProtocoIName, BrickName, BrickID,UserIteration, and TransactionLoop. Types can be Long or String. Theruntime attributes can be combined using a string function to derivevalues for brick attributes. For example, the following function createsa unique user id for each unique instance within a test plan:

userid=@string(“user”, % resourceid, % interfaceid, % userid, % id)

Using this function, the first instance of the first group of the firstinterface of the first NTR would resolve to “user0000”. The nextinstance would be “user0001” and so forth. The first instance of thefirst group of the first interface of the second NTR would resolve to“user1000”. This allows a test plan developer to create thousands ofunique attributes with just a “clicks” of a mouse.

II. Protocol Sleuthing: The following discussion sets forth examples ofprotocol sleuthing for load-testing in which the above described methodsand systems can be implemented. The following discussion is directed toa protocol sleuthing system and method for creating and implementing aplurality of synthetic users, each synthetic user implementing aplurality of synthetic transactions for cost and resource-effective loadtesting of a network server and the associated application servicesprovided thereby. The protocol sleuthing system is concomitantlyoperative to monitor each synthetic transaction and to detect and reportany error events occurring during any synthetic transaction.

Concomitant with the wide spread use of computer networks andapplication services provided by network servers in today's society isthe need to test, monitor, and evaluate the load capacity of suchnetwork servers, particularly in the context of providing applicationservices requested by users.

Effective load testing of today's networks and associated applicationservices requires a multiplicity of actual users simultaneouslygenerating continuous transactions with any particular applicationavailable through a particular network server. It is, however,economically impracticable, both from a resource standpoint and a timestandpoint, to conduct network load testing in such a manner.

Consider, for example, the resources required by an actual user inconducting one or more client-server transactions with a network server.FIG. 1 illustrates the hardware and application resources required forsuch client-server transactions. An actual user requires a computer PCthat includes hardware for inputting requests, e.g., a keyboard fortyping in the URL address of the network application to be accessed,hardware for displaying content, e.g., a monitor, and hardware forproviding a communications interface with a network server computer NSC,e.g., a modem. A client application CA that includes a graphical userinterface (GUI) with its associated drop-down menus and toolbars, e.g.,a WEB browser such as Netscape Navigator, Microsoft Internet Explorer,or Opera, is stored on the personal computer PC and provides thenecessary functionality for the user to conduct client-servertransactions with a server application SA available from the networkserver computer NSC.

FIG. 2 illustrates a typical client-server transaction, using theresources illustrated in FIG. 1, wherein the actual user utilizes theclient application CA, e.g., browser, stored on the computer PC torequest HTML pages/files from the server application SA on a networkserver computer NSC, i.e., the actual user is surfing for information onthe Internet. This client-server transaction consists of a series ofdefined protocol interactions between the browser (client application)and the server application, i.e., client requests and server responsesto such client requests, in accordance with a standard protocol such asFTP (File Transfer Protocol). For example, referring to FIG. 2, theactual user initiates the protocol interactions defined for thisparticular client-server transaction by means of a request CR1 toconnect to the network server computer NSC (by inputting the URL address“xyz” of the network server computer NSC via the client graphical userinterface GUI, e.g., by keyboard inputs or clicking on a link). Thenetwork server computer NSC issues a response SR1 granting or denyingthis client request. Once connected to the network server computer NSC,the client requests that the server application SA be opened on aspecific port number, e.g., “abc” (request CR2), and the server NSCresponds by granting this request, i.e., response SR2. Next, the clientrequests a specific file, e.g., “www.mysite.com/myfile.html” by means ofrequest CR3, and the server NSC responds by transmitting the requestedfile to the client computer PC. By means of client requests CR4 and CR5,and the server's responses thereto, i.e., responses SR4, SR5), theclient-server transaction is voluntarily terminated by the user.

At any point in these protocol interactions, however, the server NSC canrespond to any particular client request by means of an error message,i.e., the server denies a particular client request. Such a denial of aclient request may be predicated on any number of diverse events, e.g.,an authentication failure (initial request to establish a client-serverrelationship), a temporary lack of a server resource, e.g., CPUprocessing, memory, necessary to fulfill the client request, or a serverapplication processing error. By way of example, refer to FIG. 8 whichillustrates an error event occurring during a client-server transactionof the type illustrated in FIG. 2. In this particular client-servertransaction, the server NSC issues an error message, i.e., serverresponse SR2E, in response to the client request CR2. A denial of aclient request causes the client application CA to automaticallyterminate the protocol interactions, i.e., the client-servertransaction, as illustrated by client request CR5 and the correspondingserver response SR5 in FIG. 8. A premature termination of client-servertransaction due to a denial of a client request/error message from theserver NSC (or the lack of a response from the server NSC to a validclient request) is defined as an “error event”.

Client-server transactions such as the foregoing are resource intensive,the graphical interface and display engines of a browser clientapplication consuming extensive memory and CPU processing resources tosupport the graphical user interface, processing and transmission ofapplication requests, and the display of content provided by the server.A single web browser, for example, can utilize as much as 10 Megabytesof memory during a single client-server transaction.

Network load testing is currently accomplished using the clientresources and client-server transaction format described in thepreceding paragraphs in conjunction with utility macros that record theuser's interactions with the client and server applications, e.g., thesemacros record all user input (keystrokes, mouse movements and clicks)and relevant responses from the server application during theclient-server transaction. These recorded macros are subsequentlyreplayed to review and evaluate the information recorded by thesemacros. Because the user's interactions with the client and serverapplications are predominately graphical, detecting and isolating errorsis time consuming and labor intensive. Since application graphical userinterfaces do not typically include any means or mechanism for detectingerror events, the detection of error events is a visual process.

To scale the foregoing load-testing scheme to provide valid load-testingdata requires a large number of simulated users to generatestatistically-sufficient data. However, each such simulated user for theforegoing load-testing would require the entire client application,including the graphical user interface. Accordingly, to simulate 100 webusers in accordance with this load-testing scheme would require theclient computer to instantiate 100 instances of the client applicationand the network server application, which would typically overwhelm thememory resources of the client computer. Therefore, because of excessiveresource requirements, the foregoing load-scheme is not scalable to theextent necessary to generate statistically-valid data for network loadtesting.

A need exists to provide a means for load-testing network servers andthe associated application services provided thereby that is notresource intensive. Such a means should be capable of being implementedusing a single computer system. Such means should concomitantly enablemonitoring, detecting, and reporting of error events detected duringnetwork load-testing

Accordingly, a protocol sleuthing system described herein creates aplurality of synthetic users wherein each of the synthetic usersgenerates a plurality of synthetic transactions in accordance with aspecified protocol for load testing of a network server.

Also provided is a means for monitoring each of the plurality ofsynthetic transactions generated by each of the plurality of syntheticusers to detect any error events occurring during any of such synthetictransactions and reporting any error events detected during any of suchsynthetic transactions.

In a further aspect, a protocol sleuthing system can load test a networkserver that includes a computer configured to interconnect with thenetwork server, a protocol engine stored in and implemented by thecomputer and operative to generate a plurality of synthetic users, togenerate a synthetic transaction in accordance with a specifiedprotocol, and to cause each of the plurality of synthetic users tosequentially implement a plurality of the synthetic transactions withthe network server for load testing thereof, a configuration fileconnected to the protocol engine that includes variables required togenerate the synthetic transaction, information that defines thebehavior of the plurality of synthetic users implementing the synthetictransaction, and information that defines the number of synthetic usersto be created by the protocol engine, and a module that is operative tomonitor each of the plurality of synthetic transactions implemented byeach of the plurality of synthetic users with the network server, todetect any error event occurring during any of the plurality ofsynthetic transactions implemented by any of the plurality of syntheticusers, and to report any error event detected during such networktesting.

FIG. 9 illustrates one embodiment of a system 10 for protocol sleuthing.The protocol sleuthing system 10 comprises a plurality of interactivecomponents that provide the functionality necessary to create aplurality of ‘synthetic users’, to establish a client-serverrelationship and generate a sequential plurality of ‘synthetictransactions’ with a network server NSC for each ‘synthetic user’, andto monitor each ‘synthetic transaction’ for the purpose of detecting andreporting any error event occurring during each such ‘synthetictransaction’. The protocol sleuthing system 10 effectively provides awindowless background client that does not require any type of userinterface to generate ‘synthetic transactions’ with a network server,i.e., the system 10 is not resource intensive. The protocol sleuthingsystem 10 provides the capability to create and implement a large numberof ‘synthetic users’ on a single computer system, thereby providing thenecessary scalability to ensure statistically-significant load testingof network servers and their associated application services.

The protocol sleuthing system 10 has utility for network load testingbased upon client-server transactions in accordance with a standardprotocol such as HTTP (Hypertext Transfer Protocol), FTP (File TransferProtocol), SMTP (Simple Mail Transfer Protocol), POP3 (Post OfficeProtocol 3), IMAP (Internet Message Access Protocol), and NNTP (NetworkNews Transfer Protocol). In addition to client-server transactions usinga standard protocol, the protocol sleuthing system 10 also has utilityin client-server transactions using a defined protocol (“defined” beingused in the context that the protocol is documented, but not universallyor generally used/accepted, e.g., a proprietary protocol used by anenterprise in intranet or extranet transactions—contrast with ‘standard’which indicates a protocol established by general consent (or authority)as a general model or example that is universally or generallyused/accepted).

The protocol sleuthing system is implemented by means of a computer Cand comprises a protocol engine 20, a configuration file 30, and amonitoring, detecting, and reporting (MDR) module 40.

The protocol engine 20, which can be stored in and implemented by asingle computer C, is operative to generate a plurality of ‘syntheticusers’, and is further operative to cause each ‘synthetic user’ togenerate multiple, sequential ‘synthetic transactions’ with the networkserver NSC in accordance with a specified protocol. Concomitantly, theprotocol engine 20 is also operative to implement the MDR module 40 tocontinuously monitor each ‘synthetic transaction’ and to detect andreport any error event occurring during any ‘synthetic transaction’. Theprotocol engine 20 includes a first set of instructions 22, a second setof instructions 24, and a set of control instructions 26.

The first set of instructions 22 executed by the protocol engine 20establishes a client-server relationship with the network server NSC inaccordance with a specified protocol. For example, where the ‘synthetictransaction’ is a file transfer request to the network server NSC inaccordance with the File Transfer Protocol, the first set ofinstructions 22 executed by the protocol engine 20 implement the clientrequests CR1, CR2 exemplarily illustrated in FIG. 2 to establish aclient-server relationship between the computer C and the network serverNSC (see also FIG. 1). The protocol engine 20 is also operative toexecute the first set of instructions 22 to terminate the client-serverrelationship, for example, at the successful conclusion of a ‘synthetictransaction’, as exemplarily illustrated by the client requests CR4, CR5in FIG. 10. Or, as illustrated in FIG. 8, the protocol engine 20 isoperative to execute the first set of instructions to terminate theclient-server relationship upon the detection of an error event duringthe establishment of the client-server relationship, as exemplarilyillustrated by the client request CR5 in FIG. 8.

The first set of instructions 22 can also include a subset ofinstructions for retrieving any variable(s) from the configuration file30 necessary to establish the client-server relationship between thecomputer C and the network server NSC in accordance with a specifiedprotocol. Most typically, such variables would include a user or loginname and a password where the specified protocol requires an“authentication login protocol” as a prelude to establishing aclient-server relationship. A representative instruction where theestablishment of the client-server relationship requires anauthentication login protocol is ftpget(username, password) where theparameters username and password are variables stored in theconfiguration file 30 (see discussion below regarding the configurationfile 30).

One skilled in the art will appreciate that the specifics regarding thefirst set of instructions 22 executed by the protocol engine 20 dependupon the protocol required for the implementation of a particular‘synthetic transaction’. One skilled in the art will be able to generatethe first set of instructions 22 necessary to establish a client-serverrelationship with any network server in accordance with a specifiedprotocol without undue experimentation.

The second set of instructions 24 executed by the protocol engine 20accomplishes the task or tasks defined by the ‘synthetic transaction’ inaccordance with the specified protocol. For example, where the‘synthetic transaction’ is a file transfer request to the network serverNSC in accordance with the File Transfer Protocol, the second set ofinstructions 24 executed by the protocol engine 20 accomplish the taskof transferring a particular file from the network server NSC to thecomputer C. To accomplish this task, the second set of instructions 24would implement the client request CR3 exemplarily illustrated in FIG. 2to request that the network server NSC transfer a copy of the particularfile, e.g., “myfile”, to the computer C.

Where the network server NSC transfers the requested content, e.g.,“myfile”, to the computer C as part of the ‘synthetic transaction’, thesecond set of instructions 24 executed by the protocol engine 20 arefurther operative to ensure that such transferred content does notencumber the memory resources of the computer C. For example, the secondset of instructions 24 can be designed to immediately delete suchtransferred content or may be designed to direct such transferredcontent to the recycle bin of the computer C.

In accomplishing any particular task or tasks defined by the ‘synthetictransaction’ in accordance with the specified protocol, the protocolengine 20 may require one or more variables in carrying out the task ortasks defined by the ‘synthetic transaction’. The second set ofinstructions 24, therefore, includes a subset of instructions forretrieving any variables necessary in accomplishing such task or taskscomprising the ‘synthetic transaction’ from the configuration file 30.For example, where the ‘synthetic transaction’ is a file transferrequest to the network server NSC in accordance with the File TransferProtocol to retrieve a particular file as exemplarily illustrated inFIG. 2, this subset of instructions would be operative to retrievevariables identifying the directory where the particular file is stored,the filename of the particular file, and the filetype of the particularfile. An illustrative, generic example of such a retrieval instructionexecuted by the protocol engine 20 under this circumstance isftpget(directory, filename, filetype) where the parameters directory,filename, and filetype are variables stored in the configuration file30. Retrieved variables are utilized in the second set of instructions24 as required to accomplish the task or tasks defined by the ‘synthetictransaction’ in accordance with the specified protocol.

One skilled in the art will appreciate that the specifics regarding thetask or tasks accomplished by means of the second set of instructions 22executed by the protocol engine 20 depend upon the protocol specifiedfor any particular ‘synthetic transaction’. One skilled in the art willbe able to generate the second set of instructions 22 necessary toaccomplish any such task or tasks specified for any particular‘synthetic transaction’ in accordance with a specified protocol withoutundue experimentation.

The configuration file 30 comprises stored information and data(variables) for a particular ‘synthetic transaction’. The configurationfile 30 can reside in any primary or secondary storage element, e.g.,memory, cache, disk, network storage, network message, accessible to theprotocol engine 20 running on the computer C (if the configuration file30 resides in secondary storage, the protocol engine 20 would preferablymove the configuration file 30 to primary storage prior to executing thefirst set of instructions 22). The variables (data) required to generatea particular ‘synthetic transaction’ in accordance with a specifiedprotocol are stored in the configuration file 30. In addition to storageof the variables required for any particular ‘synthetic transaction’,the configuration file 30 has stored therein: (1) information thatdefines the behavior of a ‘synthetic user’ implementing the ‘synthetictransaction’; e.g., how many times the each ‘synthetic user’ is togenerate the ‘synthetic transaction’, any other actions to be taken byeach ‘synthetic user’ in conjunction with the ‘synthetic transaction’(the terminology “other actions” as used herein means a function orfunctions performed by the ‘synthetic user’ that is not part of thespecified protocol), and (2) information that defines the ‘backgroundclient’ implemented by the protocol sleuthing system 10, i.e., how many‘synthetic users’ will be generated by the protocol engine 20.

For example, with reference to a ‘synthetic transaction’ that is a filetransfer request to the network server NSC in accordance with the FileTransfer Protocol to retrieve a particular file, e.g., “myfile.html”, asexemplarily illustrated in FIG. 2, the following variables would bestored in the configuration file 30 (# denotes a comment stored inconjunction with the variable): User russ #Use this as the login namePassword secret #Use this as the password Filetype htm (or html) #Thefile is an htm (or html) file Directory / #Use the default directoryFilename myfile.html #Retrieve this file

The following illustrative information defining the behavior of the‘synthetic user’ is stored in the configuration file 30 in the contextof such a file transfer request is: Repeat 50 #Repeat the file transfer50 times WaitAfterLoop10 #Wait 10 seconds after each file transfer tosimulate think time (“other action” part of the ‘synthetic transaction’)

The following illustrative information defining the ‘background client’is stored in the configuration file 30: Clones 1000 #Create 1000instances of this synthetic user.The foregoing configuration file 30 information and data defines a‘background client’ that consists of 1000 ‘synthetic users, each‘synthetic user’ implementing a ‘synthetic transaction’ 50 times, whereeach ‘synthetic transaction’ consists of a file transfer request, e.g.,for file “myfile.html”, followed by a pause period of 10 seconds.

The protocol engine 20 executes the set of control instructions 26 toimplement the ‘background client’. In the first instance, the set ofcontrol instructions 26 are executed to create the number of ‘syntheticusers’ defined by the ‘background client’. Next, the set of controlinstructions 26 cause the protocol engine 20 to execute the first set ofinstructions 22 with respect to each ‘synthetic user’ to establish aclient-server relationship between each ‘synthetic user’ and the networkserver NSC in accordance with the specified protocol (including theretrieval of any variables required to establish the client-serverrelationship from the configuration file 30). Next, the set of controlinstructions 26 cause the protocol engine 20 to execute the second setof instructions 24 with respect to each ‘synthetic user’ to implementthe behavior of the ‘synthetic users’, i.e., accomplish the task ortasks defined by the ‘synthetic transaction’ in accordance with thespecified protocol (including the retrieval of any variables required toaccomplish any task or tasks defined by the ‘synthetic transaction’).Next, the set of control instructions 26 cause the protocol engine 20 torepeat the execution of the second set of instructions 24 with respectto each ‘synthetic user’ as defined in the configuration file 30, i.e.,to cause each ‘synthetic user’ to implement the number of ‘synthetictransactions’ defined by the configuration file 30 (e.g., 50 ‘synthetictransactions’ in the illustrative ‘synthetic transaction’ describedabove). Finally, upon completion of the second set of instructions 24,i.e., the number of ‘synthetic transactions’ defined by theconfiguration file 30 has been completed, the control instructions 26cause the protocol engine 20 to execute the first set of instructions 24to terminate the client-server relationship for each ‘synthetic user’.An illustrative example of the implementation of a ‘background client’consisting of four ‘synthetic users’ is depicted in FIG. 10.

One skilled in the art will be able to generate the second set ofinstructions 22 necessary to implement a ‘background client’ inaccordance with the description herein without undue experimentation.

The protocol engine 20 is further operative to implement the MDR module40 during the execution of the first and second sets of instructions 22,24 by the protocol engine 20 operating under the set of controlinstructions 26. For purposes of facilitating a more completeunderstanding of this aspect of the protocol sleuthing system 10, theMDR module 40 is depicted as an element separate and distinct from theprotocol engine 20. For this embodiment, the protocol engine 20 usesAPIs to implement the functionality of the MDR module 40. Alternatively,the functionality of the MDR module 40 could be implemented as anotherset of instructions stored in the protocol engine 20.

The MDR module 40 is operative to monitor each client-server interactionduring establishment of the client-server relationship between each‘synthetic user’ and the network server NSC and to monitor eachclient-server interaction between each ‘synthetic user’ and the networkserver NSC during each ‘synthetic transaction’. The MDR module 40 isfurther operative to detect any ‘error event’, i.e., error code, thatoccurs during any of the foregoing client-server interactions. Forexample, referring to FIG. 8, an exemplary ‘error event’ (as a result ofthe FTP application of the network server NSC having insufficient memoryresources to respond to the ‘synthetic user’ request CR2 to open theapplication on port abc) that occurs during the establishment of theclient-server relationship for any particular ‘synthetic user’ isillustrated. This exemplary ‘error event’ is transmitted to theparticular ‘synthetic user’ by the network server NSC as a response SR2to the ‘synthetic user’ request CR2. The MDR module 40 is operative todetect this ‘error event’ as an anomaly in the context of the expectedclient-server interactions defined by the specified protocol. Finally,MDR module 40 is operative to provide notification of this anomalousoccurrence, i.e., the ‘error event’, as well as an identification of thenetwork server NSC and any relevant context information that canfacilitate the isolation/identification of the particular application onthe network server NSC responsible for the ‘error event’ to anappropriate application for subsequent processing, e.g., an applicationstored at the network management station NMS, as exemplarily illustratedin FIG. 9. ‘Error event’ reporting can be effected via the protocolengine 20, as illustrated in FIG. 9, or can be effected directly betweenthe MDR module 40 and the network management station NMS.

FIG. 11 illustrates the differences between a client-server transactionconducted by an actual user and a client-server transaction implementedby a background client via the protocol sleuthing system and methods.Since there is no human associated with any ‘synthetic user’, there isno need for a user interface, graphic display, or permanent storage ofany content provided by a network server application utilizing theprotocol sleuthing system 10. The protocol sleuthing system 10 usessubstantially less resources in terms of memory and CPU utilization,which allows large numbers of ‘synthetic users’ to be generated via asingle client computer.

A variety of modifications and variations of the systems and methodsdescribed herein are possible.

1. A network load testing system comprising: an addressable named listmeans to enable the generation of substantially random and uniquenetwork transaction instances simulative of real network trafficpatterns, addressing means operable to address the named list means, andgenerating means, operable to communicate with the addressing means, forgenerating the substantially random and unique network transactioninstances simulative of real network traffic patterns.
 2. A network loadtesting system comprising: means to enable the generation ofsubstantially random and unique attributes to vary a population ofsynthetic user attributes.
 3. The network load testing system of claim 2wherein: the synthetic user attributes include any of URLS, hosts,security levels, authentication, ports, and headers.
 4. A network loadtesting system according to claim 1, further comprising: means to enablethe generation of substantially unique but substantially predictablesynthetic user attributes for introducing variation into ones of aseries of instances.
 5. A network load testing system accordingly toclaim 1, further comprising: means for generating network transactioninstances in accordance with a distribution that is substantially randombut representative of realistic population loads.
 6. A network loadtesting system comprising: means for generating synthetic transactioninstances, simulative of the network load presented by real users, inaccordance with a test plan containing multiple population classes, andwherein: each of the population classes contains attributes thatdescribe the behavior of each instance generated in association with thegroup.
 7. A network load testing system according to claim 6 furthercomprising: a network testing resource application for generating, basedon the test plan and the attributes contained in the population classestherein, a series of instances and a selected network protocolrepresentative of an actual load.